The present invention relates generally to computer systems, and more particularly to systems for organizing and formatting text and handwritten images on a computer display screen.
Computerized personal organizers are becoming increasingly popular. They perform such functions as keeping calendars, address books, to-do lists, etc. While these functions can be provided by conventional computer systems, they are more conveniently provided by personal organizers which are relatively inexpensive, small, and lightweight (i.e. portable). Personal organizers are available from such companies as Sharp and Casio of Japan.
A relatively new form of computer, the pen-based computer system, combines the power of a general purpose computer with the functionality and small size of a personal organizer. A pen-based computer system is typically a small, hand-held computer where the primary method for inputting data includes a "pen" or stylus. A pen-based computer system is commonly housed in a generally rectangular enclosure, and has a dual-function display assembly providing a viewing screen along one of the planar sides of the enclosure. The dual-function display assembly serves as both an input device and an output device. When operating as an input device, the display assembly senses the position of the tip of a stylus on the viewing screen and provides this positional information to the computer's central processing unit (CPU). Some display assemblies can also sense the pressure of the stylus on the screen to provide further information to the CPU. When operating as an output device, the display assembly presents computer-generated images on the screen.
The dual-function displays of pen-based computer systems permit users to operate the computers as computerized notepads. For example, graphical images can be input into the pen-based computer by merely moving the stylus across the surface of the screen. As the CPU senses the position and movement of the stylus, it generates a corresponding image on the screen to create the illusion that the stylus is drawing the image directly upon the screen, i.e. that the stylus is "inking" an image on the screen.
With suitable recognition software, the user's handwriting or "ink" can be identified as text and numeric information.
Thus, pen-based computer systems can take and retrieve notes in the user's own handwriting. Humans have been jotting notes and preparing documents in their own handwriting for centuries, but the resulting script can be difficult to interpret. Often two or more handwritten letters fade together in a word and can not be deciphered. Further, most handwriting is written with inconsistent spacing between each character of a word, each word of a line, etc. Some handwriting is so illegible even the author can not recognize his or her own notes a few hours after he or she puts them on paper. In the past, typewriters and then word-processors have been used to provide more consistent, easy to read documents. Each character keystroke from a keyboard provides a character or space of fixed dimensions and shape. The resulting text is thereby presented in a consistent easy to read format, regardless of who the author may have been. Unfortunately for most people, typing is not nearly as fast or easy as writing in their own hand.
Pen-based computers can take notes written in their own handwriting and display them in a standard font, as if they had been entered by a keyboard. However, the recognition systems used to analyze handwriting are still imperfect, and therefore fail to provide the correct recognition of some handwritten words. Nevertheless many if not most words can be recognized with high accuracy. Many recognition systems introduce a confidence ranking reflecting the confidence that the recognition software has in its own interpretation of a handwritten word. The confidence of the recognition provided depends upon several factors including whether a handwritten character looks like a known letter of the alphabet and whether a word looks like an understood word (e.g., whether it appears in a dictionary). After analyzing such factors, the recognizer knows which words are likely to be correctly identified. Recognizers are discussed in copending U.S. patent application Ser. No. 08/001,123, filed on Jan. 15, 1993, entitled METHOD AND APPARATUS FOR COMPUTERIZED RECOGNITION, and naming Pagallo et al. as inventors and Ser. No. 08/068,443, filed on May 27, 1993, entitled METHOD AND APPARATUS FOR RECOGNIZING HANDWRITTEN WORDS, and naming Beernink as inventor, both of which are incorporated herein by reference for all purposes.
When confronted with poorly recognized words, a pen-based computer system has the option of converting the poorly recognized words into standard fonts (and risking a wrong interpretation) or leaving them as handwriting. If the system leaves the poorly recognized words as handwriting, it produces a mix of word objects: some recognized text words and other unrecognized "ink" words. Unfortunately, recognized and unrecognized words are often treated quite differently by the pen-based computers. Some current systems reformat or "reflow" all recognized text words into paragraphs but do not include the unrecognized ink words. The text words in these systems are reflowed according to standard word processing protocols (e.g., wrapping text lines between two margins, cutting and pasting, etc.). The unrecognized ink words, on the other hand, are formatted independently of the text words, or sometimes are simply written on top of the text words.
Although prior pen-based computer systems with handwriting recognizers have met with some success, it would be desirable to have other systems which might be more compatible with some users' writing styles. Particularly, it would be desirable if unrecognized ink images and recognized text could be treated as equivalent for some word processing routines.